Small Models of Large Machines

In this paper, we model large support vector machines (SVMs) by smaller networks in order to decrease the computational cost. The key idea is to generate additional training patterns using a trained SVM and use these additional patterns along with the original training patterns to train a neural network. Results verify the validity of the technique. Introduction A key element in a pattern recognition system is the classifier. In nearest neighbor classifiers (NNCs) the input feature vector is compared to all the training patterns and assigned to the class to which the nearest training pattern belongs. The NNC approximates the Bayes classifier (Fukunaga 1990), but is computationally very expensive. A multilayer perceptron (MLP) classifier approximates Bayes discriminant in a least squares sense (Ruck et al. 1990) and is computationally more economic than an NNC. However, it requires many training patterns in order to generalize and it is sensitive to initial values of the weights. Support vector machines (SVMs) (Vapnik 1998) map the inputs to a large feature space, making use of classical regularization, and allowing a subset of the training patterns to be support vectors. SVMs can often develop good decision boundaries from small data sets. However, it is difficult to find the best kernel and kernel parameters (Brown et al. 2000), and the resulting structure is too large and redundant. Attempts have been made to reduce the size of the SVM (de Kruif & de Vries 2003) by deleting some patterns and by developing the Reduced Support Vector Machine (RSVM) (Lee & Huang 2007), pruning of the SVM (de Kruif & de Vries 2003) and the Relevance Vector Machine (RVM) (Tipping 2000). Unfortunately, networks resulting from these techniques are still large. In this paper, we model SVMs by much smaller machines, in order to reduce computational complexity. First, we review the MLP and the SVM. Pattern memorization in SVMs is investigated next. We then present methods for compact modeling of the SVM. Finally, we provide numerical results and conclusions. Copyright c © 2008, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. Review Of Learning Machines In this section we review MLP and SVM classifiers. Multilayer Perceptron Let {xp, tp} Nv p=1 be the training dataset where xp ∈ R N is the input vector, tp ∈ R M is the desired output vector and Nv is the number of patterns. Figure 1 depicts a feedforward MLP, having one hidden layer with Nh nonlinear units and an output layer with M linear units. For the p pattern, the j hidden unit’s net function, {netpj} Nh j=1, and activation, {Opj} Nh j=1, are